IAC-10.D2.5.7 


ARES PROJECT 
TECHNOLOGY ASSESSMENT 
- APPROACH AND TOOLS - 


Presented at the 61th International Astronautical Congress 2010 

Prague , Czech Republic 


Uwe Hueter 

Science Applications International Corporation (SAIC) 
Huntsville, AL, United States 
uwe.hueter@saic.com 

& 

Richard Tyson 

University of Alabama in Huntsville, AL, United States 
richard.w.tyson@nasa.gov 


Abstract: The paper discusses the Ares Project’s approach to technology assessment. 
System benefit analysis, risk assessment, technology prioritization, and technology readiness 
assessment will be addressed. A description of the technology readiness level (TRL) tool 
being used will be provided. 


IAC-10.D2.5.7 


ARES PROJECT TECHNOLOGY ASSESSMENT 
- APPROACH AND TOOLS - 

Uwe Hueter* & Richard Tyson** 

NASA - Marshall Space Flight Center 
Marshall Space Flight Center, AL 35812 


Abstract 


Introduction 


Technology assessments provide a status of the 
development maturity of specific technologies. 
Along with benefit analysis, the risks the 
project assumes can be quantified. Normally 
due to budget constraints, the competing 
technologies are prioritized and decisions are 
made which ones to fund. A detailed 
technology development plan is produced for 
the selected technologies to provide a roadmap 
to reach the desired maturity by the project’s 
critical design review. 

Technology assessments can be conducted for 
both technology only tasks or for product 
development programs. This paper is primarily 
biased toward the product development 
programs. 

The paper discusses the Ares Project’s 
approach to technology assessment. System 
benefit analysis, risk assessment, technology 
prioritization, and technology readiness 
assessment are addressed. A description of the 
technology readiness level tool being used is 
provided. 


* Uwe Hueter, SAIC, Senior Engineer, AIAA 
member. 

** Richard Tyson, UAH, AIAA member. 


Assessments of technologies are conducted to 
establish the risk a project may incur if the 
technology is incorporated in the design. To 
determine which technology will be inserted in 
the design, a typical approach is to conduct 
system benefit analysis to ascertain the 
enhancements the technology provides to the 
proposed concept(s). Then their current status 
is assessed. Once the technology status has 
been determined, resource estimates are 
developed for maturing the technology to the 
desired state. Normally due to budget 
constraints, technologies are prioritized to 
determine which ones to fund. For the selected 
technologies, a Technology Development Plan 
(TDP) is produced to provide a roadmap to 
reach the desired maturity by the project’s 
Critical Design Review (CDR). 

A technology readiness level (TRL) approach 
has been in use by the National Aeronautics 
and Space Administration (NASA) since the 
1980's 1 . The United States Air Force (USAF) 
began using TRL in the 1990's. Today, both 
NASA and Department of Defense (DoD) 
utilize the TRL methodology to determine both 
maturity and risk for a project. 

The abbreviated TRL definitions, along with 
the key focus areas, for each of the nine levels 
currently used are shown in Figure 1. 
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the program’s approval process that are gates to 
determine if the program should be approved to 
progress to the next phase in its development 
cycle. As shown in the figure, the TRL relates 
to these milestones and to the following major 
design reviews: 

• SRR ~ Systems Requirements Review 

• SDR ~ System Definition Review 

• PDR ~ Preliminary Design Review 

• CDR ~ Critical Design Review 

As noted in Figure 3, TRL spans the entire life 
cycle of a program and is not only applicable 
to focused technology efforts. For the PDR and 
the CDR, the systems should be matured to a 
minimum TRL of 5 and 6, respectively. The 
Ares I Project set a TRL of 6 for the PDR goal. 


Technologies are evaluated against three 
primary elements during the assessment 
process: hardware fidelity, level of integration, 
and environment fidelity. Figure 2 shows how 
these three primary elements relate to TRL. 


Depending on the maturity of the concept or 
system, the following approaches have been 
used to assess technologies: 

• Expert Opinion 

• Root Source Analysis/ValuStream™ 

• Analytic Hierarchy Process (AHP) 

• Interview 

Each of the methods will be discussed in the 
section titled, TRL Assessment Approaches. 
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Figure 2. TRL vs. Primary Elements 


A typical NASA program’s life cycle is shown 
in Figure 3. The key decision points (KDPs) 
shown in the figure are defined milestones in 



Figure 3. Typical NASA Program Life Cycle 


Level 1 Definition: Basic principles observed and reported 

(Key Focus: Basic scientific principles identified) 


Level 2 Definition: Technology concept and/or application formulated 

(Key Focus: Analysis, concept, research approach) 


Level 3 Definition: Analytical and/or experimental critical functions and/or 
characteristics proof-of-concept 

(Key Focus: Components, proof-of-concept, components, demonstration in laboratory environment) 


Level 4 Definition: Component and/or breadboard demonstrated 
in a laboratory environment 

(Key Focus: Requirements & environmentdefinition, components, breadboard, demonstration in lab. Environ.) 


Level 5 Definition: Component and/or brassboard demonstrated in a relevant environment 

(Key Focus: Component integration, interfaces, brassboard, demonstration in relevant environment) 

Level 6 Definition: System/Subsystem engineering model demonstration 
in a relevant environment 

(Key Focus: Subsystem/system integration, engineering model, demonstration in relevant environment) 


Level 7 Definition: High-fidelity functionality and scaled form/fit demonstrated 
in its operational environment 

(Key Focus: integrated system, prototype, demonstration in operational environment) 


Level 8 Definition: Final product in mission configuration qualified through 
test and evaluation 

(Key Focus: flight hardware, qualification testing, demonstration in operational environment) 


Level 9 Definition: Final product in mission configuration proven in actual flight 

(Key Focus: flight hardware, operations, actual environment) 


Figure 1. Abbreviated TRL Definitions 



The paper focuses on the Ares Project’s 
approach to technology assessment including a 
description of the technology readiness level 
(TRL) tool being used. 


Technology Assessment 


Early technology assessments provide a 
program the necessary insight regarding the 
state of maturity of the elements being 
considered in the concept(s). A typical process 
used to establish a candidate technology list is 
shown in Figure 4. Based on the mission 
requirements, the system derived requirements 
are formulated. To satisfy the system 
requirements, technologies are inserted in to the 
candidate concepts(s) to determine their 
influence in accomplishing the requirements. 
This is an iterative process as depicted by the 
circular arrows in Figure 4. The results from 
these system analyses produce a list of potential 
technologies needed to enable or enhance the 
mission. 
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Utilizing this list of technologies, an 
assessment is conducted to establish the 
programmatic impacts. The assessment process 
is shown in Figure 5. First, the current state of 
these technologies are established and 
displayed by TRL as not started, partially 
completed or fully completed (blue color scale 
shown under TRL assessment in Figure 5). 
Then the degree of difficulty (technical, 
schedule, and cost) to mature these 
technologies is estimated. Based on these 
assessments, a prioritized list of both enabling 
and enhancing technologies can be developed. 
Typically the aggregate cost of maturing all of 
the technologies exceeds the available 
resources. Normally the enabling technologies 
are funded and the enhancing technologies 
providing the best benefits are funded to a level 
commensurate with the remaining available 
resources. 

The described process’ benefits are that it 
provides a co mm on understanding of the 
programmatic risk each element contributes to 
the project. This insight helps management in 
allocating adequate resources and margins to 
make the project successful. 


The disadvantages of assessments are that it 
can be very labor intensive depending on the 
type approach used. NASA requires TRL 
assessments at each major design review 
through CDR, so adequate resources have to be 
allocated to perform this task. 


Figure 4. Establishing Technology Needs 
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TRL Assessment 

• Current Technology Status 

• Identification of Technologies Shortfalls 



I- 

Degree of Difficulty Assessment 

• Technical, Cost, and Schedule 

• How long will it take to get to TRL 6 

• How difficult will it be to get to TRL 6 

I- 

Technology Prioritization 

• Enabling Technologies 

• Enhancing Technologies 

I- 

Risk Identificotion/Ouantification 

• Mitigation Plans 

• Roadmaps 
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Figure 5. Technology Assessment Process 


process requires a minimum investment in time 
and labor. Due to the unstructured nature of the 
assessment and the strong influence of 
dominant personalities, the results tended to be 
inconsistent over the life of a program and is 
one of this approaches’ major weakness. 

Root Source Analysis/ValuStream ™ : Root 

2 

Source Analysis (RoSA) , sometimes referred 
to as the Valustream method, anticipates, 
identifies, and prioritizes knowledge shortfalls 
and assumptions that are likely to create 
significant uncertainties or risks. It is a 
systems-engineering methodology developed at 
NASA over the past ten years. By 
systematically examining critical assumptions 
and the state of the knowledge needed to 
mature the technologies to satisfy requirements. 
The methodology answers the following critical 
questions: 

• What’s been included? 

• What’s been left out? 

• How has it been validated? 

• Has the real source of the 
uncertainty/risk been identified? 

An example of a typical two-dimensional 
matrix utilized in this process is shown in 
Figure 6. 


TRL Assessment Approaches 

As mentioned in the introduction, several 
approaches are available for conducting TRL 
assessments. Brief overviews of the main 
approaches are described in the following 
paragraphs. 

Expert Opinion Approach: The first and oldest 
approach, typically used before structured 
methodologies were available, was to meet 
with a few experienced people to discuss the 
merits and establish the technology levels. This 
approach has been affectionately referred to as 
BOGSA (Bunch Of Guys Sitting Around). The 
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Figure 6. Value Stream-Example 
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The left side of the matrix is the project’s work 
breakdown structure (WBS). The engineering 
base capabilities needed to mature the 
technologies are shown across the top of the 
matrix. The cells of the matrix identify the 
shortfalls that need to be addressed. 

This approach is the most labor intensive 
approach of all the approaches discussed. It 
requires large groups of people and an 
inordinate investment in time to complete. The 
status of each technology is discussed in detail 
with specific experts advancing these 
technologies in the laboratories. It probably 
provides the most accurate assessment, 
especially in the early formulation of a project 
when the concept is not well defined and 
system analyses are limited or non-existent. 
This methodology was utilized by the precursor 
of the Ares project named the Next Generation 
Launch Technology (NGLT) project. It was 
also used when first establishing the technology 
needs for the Ares V launch vehicle, the heavy- 
lift cargo vehicle of the Ares Project. 

3 

AHP Approach : The Analytic Hierarchy 
Process (AHP) is based on the hierarchical 
decomposition of the prioritization down to the 
level at which the decision alternatives can be 
pair-wise compared for relative strength against 
the criteria. That decides the relative 
importance of technologies by comparing each 
pair of objectives and weight ranking them on a 
scale of importance, see Figure 7. 



Figure 7. Analytic Hierarchy 


Both qualitative and quantitative aspects of the 
assessment can be utilized in the process. The 
pair-wise comparisons are usually made by 
teams of subject matter experts. These 
comparisons are then translated onto a 
numerical ratio scale. The intensity of 
importance is based on the scale shown in 
Figure 8. Available software converts the 
opinions provided by the participants to 
priorities and calculates a consistency ratio that 
provides insight into the variances between 
participant’s responses. 


Intensity of 
Importance 

Description 

1 

Equal Importance 

3 

Moderate Importance 

5 

Strong Importance 

7 

Very Strong Importance 

9 

Extreme Importance 


Figure 8. Pairwise Scaling Factors 
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The AHP process can be broken down into the 
following five primary steps: 

• Hierarchical modeling of the problem 
including decision goal, the alternatives, 
and the criteria. 

• Establishing priorities among the 
groups of through judgments based on 
pairwise comparisons. 

• Calculating the overall priorities for the 
hierarchy. 

• Checking judgment consistency. 

• Making a decision based on the results. 

The process is less labor intensive than the 
RoSA approach but still requires a substantial 
labor investment involving a large group of 
people over several days of continuous 
assessment. This methodology was utilized by 
the NGLT project. 

Interview Approach: The interview approach is 
based on a Microsoft Excel based TRL 
calculator developed by Mr. William Nolte of 

4 

the Air Force Research Laboratory (AFRL) . 
The approach taken was to interview subject 
matter experts based on very structured and 
very specific set of questions to establish the 
technology level. The questions at each level of 
technology are based on the technology 
readiness level scale shown in Figure 1. The 
main parameters the questions focus on are 
requirements, hardware fidelity, scaling, tests 
completed, and test environments, see Figure 2. 
The technology assessment can be conducted 
for each of the following categories: 

• Hardware Readiness Level (TRL) 

• Software Readiness Level (SRL) 

• Manufacturing Readiness Level (MRL) 

As an example, the hardware questions for 
TRL 6 are shown in Figure 9. Each level and 
category has a set of individual tailored 


questions that are assessed by the person(s) 
being interviewed. 


1. System requirements finalized? 

2. Operating environment definition finalized? 

3. Subset of relevant environments identified that address key aspects of the final 
operating environment? 

4. M&S used to simulate system performance in an operational environment? 

5. M&S used to simulate system/subsystem engineering model/protype 
performance in the relevant environment? 

6. External interfaces baselined? 

7. Scaling requirements finalized? 

8. Facilities, GSE, STE available to support engineering model testing in the 
relevant environment? 

9. Engineering model or prototype that adequately addresses critical scaling 
issues fabricated? 

10. Engineering model or prototype that adequately addresses critical scaling 
issues tested in the relevant environment? 

11. Analysis of test results verify performance predictions for relevant 
environment? 

12. Test performance demonstratingagreement with performance predictions 
documented? 


Figure 9. TRL 6 Hardware Questions 


This process requires an acceptable amount of 
time and resource commitment. Due to the 
structured nature of this approach, it also 
provides a consistent set of parameters that 
must be considered for each assessment. 


Ares Approach to TRL Assessment 

The NASA Procedural Requirements (NPR) 
document, “NASA Program and Project 
Management Processes and Requirements”, 
designated as NPR 7120.5 establishes the 
requirements by which NASA will formulate 
and implement space flight programs and 
projects. NPR 7120.5 specifies that a 
Technology Development Plan (TDP) is to be 
base lined by KDP B and the TRL Assessments 
be base lined by KDP C (see Figure 3). 

NPR 7123.1, “NASA Systems Engineering 
Processes and Requirements”, clearly 
articulates and establishes the requirements on 
the implementing organization for performing, 
supporting, and evaluating system’s 
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engineering to ensure NASA products meet 
customers’ needs. NPR 7123.1 specifies that 
one of the entrance criteria for the SDR, PDR, 
and CDR is an updated Technology Readiness 
Assessment Report (TRAR). 

Due to the Ares I project rapid buildup; the 
project studied how the required TRL 
assessment could be conducted with minimum 
impacts to an already overworked work force. 
The approach by the Ares project was to 
minimize the impact of an assessment on the 
project without compromising the fidelity of 
the assessment. Programs whose life cycle span 
multiple years need an approach to accurately 
and consistently assess progress through use of 
a well defined set of criteria. After assessing 
the various methods available, the project 
decided on the Interview approach and the use 
of the AFRL TRL calculator. 

After initial use of the AFRL calculator, the 
Marshall Space Flight Center (MSFC) decided 
to modify the calculator because the questions 
posed were very DoD centric and did not relate 
well to the NASA mission model. The 
modifications were conducted under the 
direction of the Assistant Director for 
Technology/Chief Technologist at MSFC 5 with 
support from the Ares Project. Most of the 
questions were restructured and reordered to 
follow the process established by the NASA 
program life cycle requirements. 

The restructuring of the program was 
somewhat problematic due to the large number 
of embedded macros in the spreadsheet and no 
available instruction on how the underlying 
logic was connected. Mr. Nolte was therefore 
recruited to help in making the desired changes. 
During this conversion, a simplified Advanced 
Degree of Difficulty (AD 2 ) calculator was 
added to the program. The AD 2 calculator 
quantifies at the first order the risk associated 


with the technology 6 . It assesses technical, cost, 
and schedule risk against user defined ranges. 
The questions are primarily associated with the 
existence of facilities, tools, materials and skills 
and capabilities. The questions are answered in 
terms of the difficulty in acquiring the elements 
based on a 9 level set of criteria. The range and 
criteria for the AD 2 levels are shown in Figure 
10 . 


Degree of 
Difficulty 

Development 

Risk 

Criteria 

9 

100% 

Requires new development outside of any existing experience base. No viable 
approaches exist that can be pursued with any degree of confidence. Basic rersearch in 
key areas needed before feasible approaches can be defined. 

8 

80% 

Requires new development where similarity to existing experience base can be defined 
only in the broadest sense. Multiple development routes must be pursued. 

7 

60% 

Requires new development but similarity to existing experience is sufficient to warrant 
comparison in only a subset of critical areas. Multiple development routes must be 
pursued. 

6 

50% 

Requires new development but similarity to existing experience is sufficient to warrant 
comparison in only a subset of critical areas. Dual development approaches should be 
pursued in order to achieve a moderate degree of confidence for success. (Desired 
performance can be achieved in subsequent block upgrades with a high degree of 
confidence.) 

5 

40% 

Requires new development but similarity to existing experience is sufficient to warrant 
comparison in all critical areas. Dual development approaches should be pursued to 
provide a high degree of confidence for success. 

4 

30% 

Requires new development but similarity to existing experience is sufficient to warrant 
comparison across the board. A single development approach can be taken with a high 
degree of confidence for success. 

3 

20% 

Requires new development well within the experience base. A single development 
approach is adequate. 

2 

10% 

Exists but requires major modifications. A single development approach is adequate. 

1 

0% 

Exists with no or only minor modifications being required. A single development 
approach is adequate. 


Figure 10. AD 2 Levels-Range and Criteria 


The TRL and AD 2 calculators are intended to 
be used together to perform an integrated 
technology assessment. The AD 2 calculator 
uses the results from the TRL calculator as 
inputs. As such, they are linked together. 
However, TRL assessment can be performed 
without performing the AD 2 assessment. 

Benchmark tests utilizing SRR data and 
independent subsystem experts were conducted 
for the TRL calculator. The AD2 calculator 
was not utilized during the review process. The 
results from the modified tool were much more 
in line with the status the subsystem expert had 
assessed than what the original tool had 
predicted. Several minor adjustments were still 
needed, especially in the software area, before 
the conversion was completed. 
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The TRL calculator of the modified tool was 
utilized by the Ares project for the SDR and 
PDR reviews. The tool was also utilized to 
assess the Ares I thrust oscillation (T-O) design 
options. One novel use of this process was to 
assess the test plan of one of the T-0 options to 
determine if the planned testing were adequate 
to bring the design to a TRL of 6 by CDR. The 
results pointed out that an additional test would 
be required, resulting in a test plan change. 

Since the tool resided primarily on a personal 
computer and most assessors reside in different 
localities, a collaborative assessment could not 
be easily performed. Also, any modification to 
the tool required the originator of the tool who 
has since retired. Therefore in 2009 the Ares 
project commissioned the Science Applications 
International Corporation (SAIC) in Huntsville, 
Alabama, to convert the Excel-based tool to a 
web-based system. 

The tool coding was conducted on an internal 
SAIC server during the development process. 
After completion, a benchmark test was 
performed to assure the results matched those 
of the Excel-based tool. The benchmark test 
used the data from the Ares I PDR. Due to the 
large amount of manual manipulation required 
in the Excel-based tool, a couple of minor 
errors in the original PDR data were discovered 
during the benchmark testing. The exceptions 
were small and were primarily due to rounding 
errors. The resultant calculator minimizes 
manual manipulation and therefore is less 
prone to introducing human errors. 

The converted web-based TRL tool was 
completed November of 2009 and placed in on 
a Constellation program development server. 
Final checkout and verification that the 
program did not cause any systemic problems 
in its operating environment was completed in 
February of 2010. The web-based TRL tool is 
now considered operational. The tool is 
described in the following section. 


Ares TRL Tool Description 

The conversion of the Excel-based tool to a 
web-based system required reprogramming of 
all the logic. In that process, the lessons learned 
in the use of the Excel-based tool were applied 
to make the tool more user-friendly. Toward 
that end, as much flexibility as was possible 
was incorporated. The program now exports the 
results in either a Microsoft Excel or Word 
format for easy incorporation into a summary 
document. Also, the flexibility of adjusting the 
interview questions or selecting only those 
questions applicable to a certain program can 
readily be made. The tool has two levels of 
control, administrator and assessor. The 
flowchart of the tool structure is shown in 
Figure 11. The administrator sets up the 
assessment for his project by specifying the 
project’s WBS to the level desired for the 
assessment. There is no restriction to the level 
and, if desired, could be perform down to the 
component level. The administrator then 
assigns the assessors who should have access to 
specific WBS elements. 



Figure 11. Flowchart of Tool Structure 
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The next major step is to define how the 
assessment is to be conducted by establishing 
the question to be utilized. The basic questions, 
which have been vetted through benchmarking 
exercises during the development of the 
modified NASA calculator are part of the 
calculator's database and are provided. The 
TRL assessment can be performed for any 
combination of the following groups: TRL, 
SRL, and MRL. However, the tool has the 
capability of modifying, deleting, or adding 
additional questions if desired. The tool also 
allows for adjusting the thresholds where the 
TRL is considered either partially completed or 
fully completed based on the percentage of 
questions answered at each level. The set of 
questions for each TRL are normally not 
weighted (all questions have equal value). 
However, the tool does have the provision to 
allow weighting of each individual question. 

Once completed, the technology assessors can 
access the tool on the web and perform their 
assessment individually or collaboratively as a 
group. After all the WBS elements’ 
assessments have been completed, the tool will 
automatically aggregate the results which can 
be exported for incorporation into the final 
report. A typical WBS setup and output is 
shown in Figures 12 and 13, respectively. 
Figure 13 uses notional data. 


Element TRL Assessment AtfAssessm 


Vehicle Integration (VI) 


136905.08.02.04 


GN&C & Integrated Performance 


136905.08.02.06 


FDDR and Abort 


136905.08.02.08 


136905.08.02.06.03 


136905.08.01.02 


Forw ard Structure 


136905.08.01.02 

136905.08.01.02 

136905.08.01.02 

136905.08.01.03 
136905.08.01.03 
136905.08.01.11 
136905.08.01.11 


Pyrotechnics and Separation 


Thrust Vector Control 


5-Segment Motor-RSRMV 


Conduct | Summary | Conduct 
| Conduct | Summary | Conduct 
| Conduct | Summary | Conduct 
] Conduct | Summary | Conduct 


Conduct | Summary Conduct 
Conduct I Summary Conduct 
Conduct | Summary Conduct 
Conduct | Summary Conduct 



Technology Assessment Scoring Summary 


WBS# 

Name 

Element 

Level 

1 

2 

3 

4 

5 

6 

7 

8 

9 

xxxxxx 

Design Review 

Project 









xxxxxx.08.02 

Vehicle Integration 

System 









xxxxxx. 08.02.04 

GN&C 

Subsystem | 









xxxxxx. 08. 02. 05 

Abort 

Subsystem | 









xxxxxx. 08.02.06 

Separation 

Subsystem | 









xxxxxx. 08. 02. 07 

Software 

Subsystem | 










xxxxxx. 08.03 

First Stage 

System | 










xxxxxx. 08. 03. 01 

Forward Structure 

Subsystem | 










xxxxxx. 08. 03.02 

Parachutes 

Subsystem | 








xxxxxx. 08.03.03 

Separation 

Subsystem | 








xxxxxx. 08.03.04 

Aft Skirt 

Subsystem | 








xxxxxx. 08.03.05 

TVC 

Subsystem | 








xxxxxx. 08.03.06 

Avionics 

Subsystem | 









xxxxxx. 08. 03. 07 

Motor 

Subsystem | 









xxxxxx. 08. 03. 08 

Nozzle 

Subsystem | 










Figure 13. Example - Assessment Output 


After completion of the TRL assessment, the 
AD 2 assessment can be performed. The TRL 
results are transferred automatically into the 
AD portion of the tool. The AD assessment 
can be conducted at several levels as outlined 
in Figure 14. 


Design & Analysis 
Manufacturing 
Operations 
Test& Evaluation 


Subcateqories In Each Grou 


- Cost 

- Schedule 

- Technical Degree of Difficulty 


Figure 14. AD 2 Major Assessment Groups 


The program provides the outputs in the form 
of the most driving parameters and their 
respective values (cost, schedule, or 
development risk) for each major group being 
assessed. The items highlighted in yellow in 
Figure 15, are the resultant questions that 
contain the maximum values of each of the 
three subcategories for the set of questions in 
each of the four groups. 


Figure 12. Example - WBS Setup 
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Drivers 

Schedule 

Cost 

Technical Readiness 

Comments 

Desiqn and Analysis 





Necessary design tools 

1vrto2vr 

$10M to S20M 

80% Dev Risk 







Manufacturing 





Necessary mfg facilities 

2vrto3vr 

$10M to $20M 

30% Dev Risk 


Necessary mfa. machines 

1vrto2vr 

S20M to $50M 

40% Dev Risk 







Operations 





ODtimized for min. annual cost 

1vrto2vr 

$10M to $20M 

60% Dev Risk 







Test and Verification 





Test tooling 

Oto 6mo 

$1Mto$10M 

30% Dev Risk 



Figure 15. Example - AD 2 Output 


Also provided is a first order schedule and cost 
risk assessment. This assessment can then be 
compared to the historical parameters based on 
the norm for several programs. An example of 
the risk output table is shown in Figure 16. 



Calculated Schedule Risk Calculated Cost Risk 



Today’s Date 

1/28/2010 

Milestone 

CDR 

COR Date 

3/1/2011 

Estimated A cost 
to CDR 

80 

A budget 


remaining to COR 


Current TRL 

5 

CRh • Cost Risk, Historical 

47% 

Variartceit • Variance, Historical 

3S% 

CRe- Cost Risk, Estimated 

0% 

CR • Cost Risk 

47% 


Figure 16. Example - AD 2 Risk Output 


Summary 

The TRL approach for assessing technologies 
has been in use by NASA since the 1980's. The 
USAF began using TRL in the 1990's. Today, 
both NASA and DoD utilize the TRL 
methodology to determine both maturity and 
risk for a project. 

Early technology assessments provide a 
program the necessary insight on the state of 
maturity of the technologies being considered 


in the concept(s). It also provides a common 
understanding of the programmatic risk each 
element contributes to the project. This insight 
helps management in allocating adequate 
resources and margins to the project for the 
maturation of the selected technologies. 
Depending on the maturity of the concept or 
system, various approaches have been used to 
assess technologies. The four methods 
discussed were: 

• Expert Opinion 

• Root Source Analysis/ValuStream™ 

• Analytic Hierarchy Process (AHP) 

• Interview 

Ares chose the Interview approach for 
conducting the required technology 
assessments over the life cycle of the project. 
The benefits of this approach were the 
structured nature of the methodology, the 
accuracy of the results verified through 
benchmark testing, and acceptable manpower 
expenditure for conducting the assessment. 

MSFC modified the AFRL calculator 
developed to better relate to the NASA mission 
model. The modified tool was utilized by the 
Ares project during the SDR, and PDR. The 
tool was also utilized to assess the Ares I thrust 
oscillation (T-O) design options. One novel use 
of this process was to assess the test plan of one 
of the T-0 design options to determine if the 
planned testing were adequate to bring the 
design to a TRL of 6 by CDR. The results 
indicated that an additional test would be 
required, resulting in a test plan change. 

Since the tool resided primarily on a personal 
computer and most assessors reside in different 
localities, a collaborative assessment could not 
be easily performed. Also, any modification to 
the tool required the originator of the tool who 
has since retired. Therefore in 2009 the Ares 
project commissioned the Science Applications 
International Corporation (SAIC) in Huntsville, 
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Alabama, to convert the Excel-based tool to a 
web-based system. 

The converted web-based TRL tool has been 
named Technology Readiness Assessment Tool 
(TRAT) and was completed November of 2009 
and placed in on a Constellation program 
development server. Between November 2009, 
and March of 2010, additional features were 
installed to make it more user-friendly. Final 
checkout and verification that the program did 
not cause any systemic problems in its 
operating environment was completed in 
February of 2010. The web-based TRL tool is 
now considered operational. 

Based on Ares’ experience, the time per 
subsystem assessment ranged from 15 minutes 
to 45 minutes, depending on whether this was 
an updated or a first time assessment. 

Generally, the assessments were conducted 
with a minimum number of subject matter 
experts (subsystem engineers in the engineering 
department). Each subsystem was scheduled 
separately to minimize unproductive time and 
extraneous discussions. Although initially some 
resistance was encountered during the TRL 
assessment process, the procedure was more 
widely accepted once the process had been 
experienced. Eventually the process was even 
utilized for assessments outside of the required 
reviews. 
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Observations 



TRL is for technology programs only 
TRL assessment is duplicate to program reviews 


Reality 

TRL covers the life span of a development program 

TRL assessments actually provide additional data to 
program 

Develops resource requirements 
Identifies Risks 
Tracks progress 

Provides record for external auditing 
Required by NASA 


TUB 
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TRL Assessment 



• Current Technology Status 

* Identification of Technologies Shortfalls 
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Advanced Degree of Difficulty Assessment 

* Technical, Cost, and Schedule 

* How long will it take to get to TRL 6 

* How difficult will it be to get to TRL 6 

* Major Groups 

- Design & Analysis 

- Manufacturing 

- Operations 

- Test & Evaluation 


Technology Prioritization 


• Enabling Technologies 

* Enhancing Technologies 



Risk Identification 


• Mitigation Plans 

* Roadmaps 
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Technology Plan j 

• Technology Roadmaps 

• Critical Technologies 
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ent Methods 


Expert^ Opinion 


Analytic Hierarchy Process (AHP) 


The oldest approach 

- Meet with a few experienced people 

* Establish the technology levels 

* Bunch of Guys Sitting Around (BOGSA) 

- Unstructured assessment 
Inconsistent results over lifetime of 
program 

Minimum Resource Requirements 

- Time 

- Manpower 


Root Source^ Anaiysis/VaiuStream ™ 


Hierarchical modeling of the problem 
including decision goal, the alternatives, and 
the criteria 

Establishing priorities based on pair-wise 
comparisons 

- Relative strength against the criteria 

- Comparison based on a scale of importance 

- Calculating the overall priorities for the 
hierarchy 

Good Accuracy 

Medium Resource Requirements 

- Time 

- Manpower 


Interview Process 



Assessment at the technologist level 

- Requirements vs. capabilities 

Most accurate assessment 

- Especially in the early formulation of a 
project 

* Concept is not well defined 
' System analyses are limited or non- 
existent 

Maximum Resource Requirements 

- Time 

- Manpower 


* Interview subject matter experts, normally 
subsystem engineering experts 

- Based on very structured approach 

- Specific set of questions 

* Good Accuracy 

* Medium-to-Low Resource Requirements 
(Between AHP & Expert Opinion) 

- Time 

- Manpower 
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Technology Readiness Assessment \ 




Technology Readiness Level (TRL): Levels 1-9 

• Manufacturing Readiness Level (MRL): Levels 3-9 

• Software Readiness Level (SRL): Levels 1-9 



Advanced [ Degree of Difficulty (AD 2 ) 




Major Groups 

- Design & Analysis 

• Manufacturing 

• Operations 

• Test & Evaluation 




Subcategories In Each Group 

- Cost 

- Schedule 

- Technical Degree of Difficulty 


owclmrtfof the Tool Structure 
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Technology Readiness Assessment^ 


Technology AssessmentScoring Summary 


WBS# 



Level 



1 

2 

3 

4 

5 

6 

7 

8 

9 

xxxxxx 

Deskpi Review 

Project 










xxxxxx DR 02 

Vehicle Int elation 

System 










xxxxxx. 06.0Z04 

GN&C 

Siiisystem 








xxxxxx. 08.02.05 

Abort 

Siiisystem 

■■ 






xxxxxx. DR.DZD6 

Separation 

Siiisystem 

mmjz 






xxxxxx. 08.0Z07 

Software 

Subsystem 







xxxxxx DR 03 

Fast Stage 

System 







xxxxxx.08.03.01 

FoiwanJ St act Lie 

Siiisystem 






xxxxxx. OR 03.02 

ParactMJtes 

Siiisystem 






xxxxxx. OR 03.03 

Separation 

SiAsystem 






xxxxxx. 08. 03.04 

Art Skit 

Siiisystem 






xxxxxx. OR 03.05 

TVC 

Siiisystem 






xxxxxx. OR 03.06 

Avionics 

Siiisystem 






xxxxxx. OB. 03.07 

Motor 

Siiisystem 

■■ 





xxxxxx. OB. 03.06 

Nozzle 

Siiisystem 







Advanced [ Degree of Difficulty (AD 2 ) 


■ ■ 

Drivers 

Schedule 

Cost 

Technical 

Readiness Comments 


Manufacturing 


Necessary deskyitoob 


Necessary mlg faciBties 


Necessary rnfci- machines 


Operations 


Optimized for min. annual cost 


Test and Verification 


Testtoolin 


1yrto2yr 


2yrto 3yr 


1yrto2yr 


lyrto^r 


0to6mo 


SlOMtoSZOM 


SlGMto S20M 


S2GMto S50M 


SlGMto$20M 


SlMtoSlOM 


80% Dev Risk 


30% Dev Risk 


40% Dev Risk 


60% Dev Risk 


30% Dev Risk 


Calculated Schedule Risk 


Today's Date 

1/28/2010 

Milestone 

CDR 

CDR Date 

3/1/2011 

Esti mated ti me 
to achieve CDR 

0 months 

Actual ti me to CDR 

14 months 

Current TRL 

5 

Historical average 
program ti me 
from TRL 5 to CDR 

35 

Variance h - Variance, Historical 

7% 

SRh - Prog. Sched. 
Risk, Historical 

150% 

SRe - Technology Sched. 
Risk, Esti mate 

0% 

SR - Schedule Risk 

150% 


Calculated Cost Risk 


Today's Date 

1/28/2010 

Milestone 

CDR 

CDR Date 

3/1/2011 

Esti mated A cost 
to CDR 

$0 

A budget 
remaining to CDR 

$0 

Current TRL 

5 

CRh - Cost Risk, Historical 

47% 

Variance h - Variance, Historical 

35% 

CRe- Cost Risk, Estimated 

0% 

CR - Cost Risk 

47% 




The TRL approach for assessing technologies 

- Used by NASA since the 1980's, and by the USAF since the 1990's 

- Today, both NASA and DoD utilize the TRL methodology to determine 
both maturity and risk 

- Depending on the maturity of the concept or system, various 
approaches have been used to assess technologies: Expert Opinion, 
Root Source Analysis/ValuStream™, Analytic Hierarchy Process (AHP), 
and Interview Process 

Ares chose the Interview Process for conducting the required 

technology assessments over the life cycle of the project. 

- Structured nature of the methodology, accuracy of the results, and 
acceptable manpower expenditure 

Web-based tool developed by SAIC 

- Allow collaborative assessment 

- The tool has been named, Technology Readiness Assessment Tool 
(TRAT) 

- TRAT is now considered operational 

Based on Ares’ experience 

- Assessment time per subsystem ranged from 15 minutes (for 
reassessments) to 45 minutes (for first assessment) 

- Manpower was acceptable for conducting the assessments 


